Constrained Stochastic State Estimation of Deformable 1D Objects: Application to Single-view 3D Reconstruction of Catheters with Radio-opaque Markers

International audienceMinimally invasive fluoroscopy-based procedures are the gold standard for diagnosis and treatment of various pathologies of the cardiovascular system. This kind of procedures imply for the clinicians to infer the 3D shape of the device from 2D images, which is known to be an ill-posed 10 problem. In this paper we present a method to reconstruct the 3D shape of the interventional device, with the aim of improving the navigation. The method combines a physics-based simulation with non-linear Bayesian filter. Whereas the physics-based model provides a prediction of the shape of the device navigating within the blood vessels (taking into account non-linear interactions be-15 tween the catheter and the surrounding anatomy), an Unscented Kalman Filter is used to correct the navigation model using 2D image features as external observations. The proposed framework has been evaluated on both synthetic and real data, under different model parameterizations, filter parameters tuning and external observations data-sets. Comparing the reconstructed 3D shape with a known ground truth, for the synthetic data-set, we obtained average values for 3D Hausdorff Distance of $0.81 ± 0.53 mm$, for the 3D mean distance at the segment of $0.37 ± 0.17$ mm and an average 3D tip error of $0.24 ± 0.13 mm$. For the real data-set,we obtained an average 3D Hausdorff distance of $1.74 ± 0.77 mm$, a average 3D mean distance at the distal segment of 0.91 ± 0.14 mm, an average 3D error on the tip of $0.53 ± 0.09 mm$. These results show the ability of our method to retrieve the 3D shape of the device, under a variety of filter parameterizations and challenging conditions: uncertainties on model parameterization, ambiguous views and non-linear complex phenomena such as stick and slip motions

Constrained Stochastic State Estimation for 3D Shape Reconstruction of Catheters and Guidewires in Fluoroscopic Images

Minimally invasive fluoroscopy-based procedures are the gold standard for diagnosis and treatment of various pathologies of the cardiovascular system. This kind of procedures imply for the clinicians to infer the 3D shape of the device from 2D images, which is known to be an ill-posed problem. In this paper we present a method to reconstruct the 3D shape of the interventional device, with the aim of improving the navigation. The method combines a physics-based simulation with non-linear Bayesian filter. Whereas the physics-based model provides a prediction of the shape of the device navigating within the blood vessels (taking into account non-linear interactions between the catheter and the surrounding anatomy), an Unscented Kalman Filter is used to correct the navigation model using 2D image features as external observations. The proposed framework has been evaluated on both synthetic and real data, under different model parameterization, filter parameters tuning and external observations data-sets. Comparing the reconstructed 3D shape with a known ground truth, for the synthetic data-set, we obtained an average 3D Hausdorff distance of 0.07 ± 0.37 mm; the 3D distance at the tip equal to 0.021 ± 0.009 mm and the 3D mean distance at the distal segment of the catheter equal to 0.02 ± 0.008 mm. For the real data-set, the obtained average 3D Hausdorff Distance was of 0.95 ± 0.35 mm, the average 3D distance at the tip is equal to 0.7 ± 0.45 mm with an average 3D mean distance at the distal segment of 0.7 ± 0.46 mm. These results show the ability of our method to retrieve the 3D shape of the device, under a variety of filter parameterizations and challenging conditions: errors on the friction coefficient, ambiguous views and non-linear complex phenomena such as stick and slip motions

Constrained Stochastic State Estimation of Deformable 1D Objects: Application to Single-view 3D Reconstruction of Catheters with Radio-opaque Markers

International audienceMinimally invasive fluoroscopy-based procedures are the gold standard for diagnosis and treatment of various pathologies of the cardiovascular system. This kind of procedures imply for the clinicians to infer the 3D shape of the device from 2D images, which is known to be an ill-posed 10 problem. In this paper we present a method to reconstruct the 3D shape of the interventional device, with the aim of improving the navigation. The method combines a physics-based simulation with non-linear Bayesian filter. Whereas the physics-based model provides a prediction of the shape of the device navigating within the blood vessels (taking into account non-linear interactions be-15 tween the catheter and the surrounding anatomy), an Unscented Kalman Filter is used to correct the navigation model using 2D image features as external observations. The proposed framework has been evaluated on both synthetic and real data, under different model parameterizations, filter parameters tuning and external observations data-sets. Comparing the reconstructed 3D shape with a known ground truth, for the synthetic data-set, we obtained average values for 3D Hausdorff Distance of $0.81 ± 0.53 mm$, for the 3D mean distance at the segment of $0.37 ± 0.17$ mm and an average 3D tip error of $0.24 ± 0.13 mm$. For the real data-set,we obtained an average 3D Hausdorff distance of $1.74 ± 0.77 mm$, a average 3D mean distance at the distal segment of 0.91 ± 0.14 mm, an average 3D error on the tip of $0.53 ± 0.09 mm$. These results show the ability of our method to retrieve the 3D shape of the device, under a variety of filter parameterizations and challenging conditions: uncertainties on model parameterization, ambiguous views and non-linear complex phenomena such as stick and slip motions

Constraint-Based Simulation for Non-Rigid Real-Time Registration

International audienceIn this paper we propose a method to address the problem of non-rigid registration in real-time. We use Lagrange multipliers and soft sliding constraints to combine data acquired from dynamic image sequence and a biomechanical model of the structure of interest. The biomechanical model plays a role of regulariza-tion to improve the robustness and the flexibility of the registration. We apply our method to a pre-operative 3D CT scan of a porcine liver that is registered to a sequence of 2D dynamic MRI slices during the respiratory motion. The finite element simulation provides a full 3D representation (including heterogeneities such as vessels, tumor,. . .) of the anatomical structure in real-time

3D Physics-Based Registration of 2D Dynamic MRI Data

International audienceWe present a method allowing for intra-operative targeting of a specific anatomical feature. The method is based on a registration of 3D pre-operative data to 2D intra-operative images. Such registration is performed using an elastic model reconstructed from the 3D images, in combination with sliding constraints imposed via Lagrange multipliers. We register the pre-operative data, where the feature is clearly detectable, to intra-operative dynamic images where such feature is no more visible. Despite the lack of visibility on the 2D MRI images, we are able both to determine the location of the target as well as follow its displacement due to respiratory motion

Atlas-based Transfer of Boundary Conditions for Biomechanical Simulation

International audienceAn environment composed of different types of living tissues (such as the abdominal cavity) reveals a high complexity of boundary conditions, which are the attachments (e.g. connective tissues, ligaments) connecting different anatomical structures. Together with the material properties, the boundary conditions have a significant influence on the mechanical response of the organs, however corresponding correct me- chanical modeling remains a challenging task, as the connective struc- tures are difficult to identify in certain standard imaging modalities. In this paper, we present a method for automatic modeling of boundary con- ditions in deformable anatomical structures, which is an important step in patient-specific biomechanical simulations. The method is based on a statistical atlas which gathers data defining the connective structures at- tached to the organ of interest. In order to transfer the information stored in the atlas to a specific patient, the atlas is registered to the patient data using a physics-based technique and the resulting boundary conditions are defined according to the mean position and variance available in the atlas. The method is evaluated using abdominal scans of ten patients. The results show that the atlas provides a sufficient information about the boundary conditions which can be reliably transferred to a specific patient. The boundary conditions obtained by the atlas-based transfer show a good match both with actual segmented boundary conditions and in terms of mechanical response of deformable organs

Silhouette-based Pose Estimation for Deformable Organs Application to Surgical Augmented Reality

International audience— In this paper we introduce a method for semi-automatic registration of 3D deformable models using 2D shape outlines (silhouettes) extracted from a monocular camera view. Our framework is based on the combination of a biomechanical model of the organ with a set of projective constraints influencing the deformation of the model. To enforce convergence towards a global minimum for this ill-posed problem we interactively provide a rough (rigid) estimation of the pose. We show that our approach allows for the estimation of the non-rigid 3D pose while relying only on 2D information. The method is evaluated experimentally on a soft silicone gel model of a liver, as well as on real surgical data, providing augmented reality of the liver and the kidney using a monocular laparoscopic camera. Results show that the final elastic registration can be obtained in just a few seconds, thus remaining compatible with clinical constraints. We also evaluate the sensitivity of our approach according to both the initial alignment of the model and the silhouette length and shape

Domain Decomposition for real time Simulation of needle insertion

International audienceOur goal is to develop robotized needle insertion for drug delivery in small animals. We control the robot with a real-time Finite Element simulation that provides accurate models of the deformable environment. To predict the deformations we need to solve a contact problem which is known to be time consuming. To reduce the computational time we use the domain decomposition method: the FE mesh is split in several domains in order to extract paral-lelism for GPU computing and to concentrate the computation time around the needle

Domain Decomposition for real time Simulation of needle insertion

International audienceOur goal is to develop robotized needle insertion for drug delivery in small animals. We control the robot with a real-time Finite Element simulation that provides accurate models of the deformable environment. To predict the deformations we need to solve a contact problem which is known to be time consuming. To reduce the computational time we use the domain decomposition method: the FE mesh is split in several domains in order to extract paral-lelism for GPU computing and to concentrate the computation time around the needle

Simulation intéractive guidée par l'image pour la chirurgie endovasculaire

Les procédures mini-invasives basées sur la fluoroscopie représentent la référence pour le diagnostic et le traitement de diverses pathologies du système cardiovasculaire. Durant ce type de procédures, les cliniciens sont obligés de déduire l’environnement anatomique, ainsi que la forme de l’instrument chirurgicale, à partir d'images bidimensionnelles. Ce manque de perception de profondeur, qui est ultérieurement complexifié par l’environnement anatomique où de multiples organes et structures se superposent, a été identifié comme l’un des facteurs les plus importants affectant les performances cliniques. Plusieurs méthodes ont été proposées pour améliorer la visualisation des images fluoroscopiques 2D, ce qui permettrait d’améliorer la vision globale du clinicien et, avec cela, la réussite finale de la procédure. Une approche largement utilisée consiste à créer une reconstruction virtuelle 3D de la scène chirurgicale, et la combiner avec les images fluoroscopiques 2D pour avoir une vision augmentée. De manière générale, ce type de méthodes permettent de reconstruire la forme 3D de l’instrument chirurgicale (et / ou les structures anatomiques) en combinant : des a priori sur la forme et le comportement du dispositif avec des observations externes, qui fournissent une information incomplète sur sa configuration actuelle. Après avoir mis en évidence les limites des méthodes existantes, notre objectif était de développer une méthode qui : 1. fournit une bonne description de la forme et du comportement de l’instrument chirurgicale ; en particulier en modélisant les interactions non-rigides qui peuvent se produire avec l'anatomie environnante et les phénomènes non linéaires, tels que les contacts non-glissants ; 2. se base uniquement sur l’utilisation d’images fluoroscopiques monoculaires, sans qu'il soit nécessaire d'intégrer des capteurs externes sur le dispositif chirurgicale ; 3. prend en compte et permet de compenser les incertitudes qui pourraient exister sur le paramétrage du modèle et le bruit affectant les observations externes ; 4. est compatible avec les calculs en temps réel ; Nous avons d'abord proposé une approche purement déterministe, où les informations projectives des images fluoroscopiques sont intégrées au modèle sous forme de contraintes mécaniques. Malgré les bons résultats, la méthode proposée ne permet pas de prendre en compte des phénomènes fortement non-linéaires. De plus, les erreurs sur le modèle de navigation et le bruit sur les observations externes ne sont pas prises en compte. Pour les raisons ci-dessus, nous avons développé une nouvelle approche stochastique. Compte tenu de la difficulté du problème de reconstruction 2D-3D, la forme 3D du dispositif interventionnel peut être considérée comme une variable aléatoire. Cette variable est décrite, en même temps, par un modèle de prédiction, qui donne une description du comportement de la variable dans le temps et qui est caractérisé par des incertitudes, et des observations externes, qui sont partielles et affectées par du bruit. En particulier, cette thèse vise à développer une nouvelle approche où la simulation basée-physique de la navigation du cathéter est combinée à des observations 2D externes à travers une méthode bayésienne. Alors que le modèle basé-physique fournit une prédiction de la forme de l’instrument dans les vaisseaux sanguins (en tenant compte des interactions avec l'anatomie environnante), un filtre de Kalman Unscented est utilisé pour corriger la navigation en utilisant des informations 2D, extraites d'images fluoroscopiques. La méthode proposée a été évaluée à la fois sur des données synthétiques et réelles. A la fin de notre travail, nous présentons et analysons les limites actuelles de notre méthode, en proposant des solutions possibles, ainsi que quelques perspectives pour des futures applications.Minimally invasive fluoroscopy-based procedures are the gold standard for diagnosing and treating various pathologies of the cardiovascular system. With this kind of procedures, clinicians have to infer the 3D shape of the device from 2D images. Such lack of depth perception, combined with a dense environment of overlaying anatomical structures, has been identified as one of the major factors affecting clinical performances. Several methods have been proposed to enhance the visualization of 2D fluoroscopic images, which could improve the clinician’s global insight and consequently the positive outcomes of the procedures. A widely used approach is to create a 3D reconstruction of the surgical scene to be combined with 2D fluoroscopic images, in order to have an augmented view. In general, this kind of methods aims at retrieving the 3D shape of the device (and or anatomical structures) by combining some priors on the shape and the behaviour of the device, with external observations, providing some incomplete information on its current state. After highlighting the limitations of the existing 2D-3D reconstruction methods, our objective was to develop a method that: 1. provides a good description of both the shape and the behavior of the device; taking into account non-rigid interactions with the surrounding anatomy and non-linear phenomena (e.g. non-sliding contacts); 2. solely relies on monocular 2D fluoroscopic images, without the need to embed any external sensors onto the interventional device; 3. takes into account and compensates the uncertainties which might exist on model parameterization and the noise affecting external observations; 4. is compatible with real-time computations; We first proposed a purely deterministic approach, where projective information from 2D fluoroscopic images is integrated to the model as mechanical constraints. Despite the good results, the proposed method is not able to take into account non-linear phenomena such as stick and slip transitions. In addition, errors on both the navigation model and external observations are not taken into account. For the above reasons, we designed a new stochastic approach. Given the ill-posedness of the 2D-3D reconstruction problem, the 3D shape of the interventional device can be seen as a random variable. Such variable is described, at the same time, through an process model, which provides a description of the variable through time and it is affected by some uncertainties, and some external observations, which can provide some partial information on its current configuration and are affected by noise. In particular, this thesis aims to develop a novel approach, where a Bayesian approach is used to combine a constrained physics-based simulation of the catheter navigation, with external 2D observations extracted from 2D fluoroscopic images. Whereas the physics-based model provides a prediction of the shape of the navigation device navigating the blood vessels (taking into account non-linear interactions between the catheter and the surrounding anatomy), an Unscented Kalman Filter is used to correct the navigation model using 2D features, extracted from fluoroscopic images, as external observations. The proposed method has been evaluated on both synthetic and real data. Lastly, we present and analyse the current limitations of our method, proposing possible solutions, along with some perspectives for future works and applications